The Composite Cosine Transform on the Stiefel Manifold 
and Generalized Zeta Integrals 

E. Ournycheva and B. Rubin 

Abstract. The A-cosine transform on the unit sphere is denned by 
(T x f)(u)=[ f(v)\v -u\ x dv, uGS"' 1 , 

Js«-i 

and has many applications. We introduce a new integral transform T x f,\ £ 
C m , which generalizes the previous one for functions on the Stiefel and Grass- 
mann manifolds. We call it the composite cosine transform, by taking into 
account that its kernel agrees with the composite power function of the cone 
of positive definite symmetric matrices. Our aim is to describe the set of all 
A £ C m for which T A is injective on the space of integrable functions. We 
obtain the precise description of this set in some important cases, in particu- 
lar, for A-cosine transforms on Grassmann manifolds. The main tools are the 
classical Fourier analysis of functions of matrix argument and the relevant zeta 
integrals. 



1. Introduction 

The classical cosine transform (also known as the Blaschke-Levy transform) on 
the unit sphere S n ~ x in M™ is defined by 

(1.1) (Tf)(u)= J f(v)\vu\dv, ueS n -\ 

S n-1 

where / is an integrable even function on S n , and v ■ u is the usual inner product. 
This transform and its generalization 

(1.2) (T X f)(u)= J f(v)\vu\ x dv, 

arise in diverse areas of mathematics, in particular, in PDE, the Fourier analysis, 
integral geometry, and the Banach space theory; see |Es |. |Ga| . jGHlj . |GG| . |Koj . 
|Rul| - [Ru3j . |Saj . [Schnj . |Sej . Operators l|1.2fl have been investigated in detail 
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using the following two approaches. The first one employs the Fourier transform 
technique |Se| . jKoj . |Rulj . and relies on the formula 

(!- 3 ) /^Sl eiX ^ = c ^ly| A ( rA /)(||), c x . n = const, 

which should be properly interpreted. The second approach is based on decompo- 
sition in spherical harmonics. Namely, by the Funk-Hecke formula, 

(1.4) T x P k = c^ k {X)P k , 




for each homogeneous harmonic polynomial Pk(%) of even degree k restricted to the 
unit sphere; see |Rul| - |Ru3) . |Sa| . The Fourier-Laplace multiplier /life (A) provides 
complete information about properties of T x . 

In the last two decades a considerable attention was attracted to generalizations 
of T and T x for functions on the Grassmann manifold G„ jrra of m-dimensional linear 
subspaces of K™. We recall, that if rj 6 G n , m , £, € G n ,i, I > m, and [77^] is the 
m-dimensional volume of the parallelepiped spanned by the orthogonal projection 
of a generic orthonormal coordinate frame in rj onto £, then, by definition, 

(i-6) (T x m)= I mmfdv 

G„, m 

(we use the same notation as in (|1.2ll 'l. For I > m, the operator l|1.6[) represents the 
composition of the similar one over G ni i and the corresponding Radon transform 
acting from G„ jm to G n ,i (see, e.g., PP, |GR| ). Thus injectivity results for T x in 
this case can be easily derived from those for the Radon transform (see |GR| and 
references therein) and the case I — m. Owing to this, in the following we assume 
I = m. This case bears the basic features of the operator family (|1.6I) . 

The investigation of operators i|1.6|) for A = 1 was initiated in stochastic ge- 
ometry (processes of fiats) by Matheron Matl , Mat2| who conjectured that T 1 
is injective as well as its prototype (|l.lf) . Matheron's conjecture was disproved by 
Goodey and Howard GHl| who used the idea of Gluck and Warner |GWj to in- 
terpret the Grassmann manifold G4.2 as the direct product S 2 x S 2 of 2-spheres. 
For higher dimensions, the result then follows by induction. Operators T x for 
A = 0, 1, 2, . . . were studied in GH2, p. 117], where, by using reduction to G^^, it 
was proved that T x is non- injective for such A; see also |GHR| . |Spl| , |Sp2| , |Gr| . 
The range of the A-cosine transform was studied by Alesker and Bernstein |AB| 
for A = 1 and by Alesker PQ for all complex A, who invoked deep results from the 
representation theory. 

In the present article we develop a new approach to operators T x . Our argu- 
ment differs essentially from that in the cited papers. For technical reasons, we 
prefer to deal with 0(m) right-invariant functions on the Stiefcl manifold V n _ rn of 
orthonormal m-frames rather than with functions on G„,m- This does not change 
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the essence of the matter and leads to the following equivalent definition: 

(1.7) (T A /)(u) = J f(v)\det(v'u)\ x dv, u g V n , m , 

V n>m 

where " ' " stands for the transposed matrix, and the product v'u is understood in 
the sense of matrix mulilication. Then we regard (|1.7|) as a member of the more 
general analytic family 

(1.8) (T x f)(u) = J f{v){u'vv'u) x dv, ueV n , m , 

where A = (Ai,...,A m ) g C m and (-) A denotes the composite power function 
of the cone of positive definite m x to matrices; see Section 2.2. We call T x f 
the composite cosine transform of a function / on V n . m - The general intention is 
to obtain a higher-rank analog of the formula 1)1. 4[l . evaluate the corresponding 
multiplier /it (A) explicitly in terms of gamma functions, and use it for examination 
of T x . The particular case A = (1, . . . , 1) corresponds to Matheron's operator. We 
do not realize this project in full generality here and leave this work for future 
publications. However, our approach enables us to obtain the precise description of 
those A for which T x is injective in the following important cases (a) 2m < n, A = 
(Ai, . . . , A m ) e C m , and (b) Ai = • • • = A m = A g C, provided that T x f and T x f 
exist in the usual Lebesgue sense as absolutely convergent integrals. 

The essence of our approach is that we apply the classical Fourier transform 
technique to obtain higher-rank analogs of 1)1.3)1 and 1)1. 4[l . In the first case we 
assume / to be an arbitrary integrable function on V n>m , and in the second one 
Pk stands for the restriction to V n _ m of the corresponding 0{m) right-invariant 
dcterminantally homogeneous harmonic polynomial on the space of n x to matrices. 
Different aspects of harmonic analysis based on implementation of such polynomials 
were studied in [Herzj . |Str| . |TT| . and in a series of publications related to group 
representations. In the present article we obtain a higher-rank copy of 1)1.4)1 with the 
multiplier /ifc(A) explicitly expressed in terms of the gamma functions associated 
with the cone of positive definite to x to matrices. In the particular case Ai = ■ • • = 
A m the main result reads as follows. 

Theorem 1.1. Let n > to > 2, / g L x (G n>m ). Then (T A .f)(£) is finite for 
almost all £ g G n ,m if o,nd only if Re A > —1. For such X, the operator T A is 
injective on i 1 (G njm ) if and only if A ^ 0, 1, 2, ... . 

In particular, we show that if A is a non-negative integer and 2m < n, then 
T A , having been written in the form 1)1.7)) . annihilates all 0(m) right-invariant 
determinantally homogeneous harmonic polynomials Pk (x) of degree k > Re A + 
to — 1. 

A by-product of our investigation is a functional equation for the generalized 
zeta integrals with additional "angle component" f(v), v g V n ,m\ see 1)3.1) 1. 1)3.211 . 
This result is of independent interest. The main references related to zeta integrals 
can be found in |BSZ , FK , Kh2 , |Ru4| . The equation obtained below is, 
in fact, far-reaching higher-rank modifications of 1)1.3)1 in the language of Schwartz 
distributions. In cited papers they were obtained for / = 1. The case f = Pk was 
considered in jClj in the context of Jordan algebras. The proof presented below is 
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much simpler than that in |C1| (adapted to our case) and employs the idea from 
|Kh2j to derive the result for A € C from the more general one for A £ C m . 

The paper is organized as follows. Section 2 contains the necessary background 
material. In Section 3 we introduce the generalized zeta integrals. Section 4 plays 
the central role in the article and establishes connection between zeta integrals and 
composite cosine transforms. The results of Section 4 are applied in Section 5 to 
study injectivity of the composite cosine transform. 

Acknowledgement. We are thankful to Dr. S.P. Khekalo for sharing with us 
his results |Kh2] , 

2. Preliminaries 

We establish our notation and recall basic facts that will be used throughout 
the paper. The main references are |FK| . |Gij . |OR| . |T). 

2.1. Notation. Let 5!Jt n , m be the space of real matrices x — (xij) having n 
rows and m columns. We identify ^Ol n ,m with the real Euclidean space M. nm and set 
dx = n™=i Ilj=i ^ x i,j f° r the Lebesgue measure on 9Jl n , m - If n > m, then SD? n m 
stands for the set of all matrices x £ 9Jl n . m of rank m. This set has a full measure 
in 9Jt, im . In the following, x' denotes the transpose of x, I m is the identity m x m 
matrix, stands for zero entries. Given a square matrix a, we denote by \a\ the 
absolute value of the determinant of a, and by tr(a) the trace of a, respectively. 

Let 17 = Vm be the cone of positive definite symmetric matrices r = (ri,j) mxm 
with the elementary volume dr = Yii<j ^iji an d let 17 be the closure of 17, that is 
the set of all positive semi-definite m x m matrices. For r £ 17 (r £ 17) we write 
r > (r > 0). Given si and S2 in f2, the inequality si > S2 means si — S2 £ fi. If 
a £ f2 and 6 £ 17, then J" f(s)ds denotes the integral over the set 

{s : s £ O, a < s < b\ = {s : s — a £ fi, b — s 6 f2}. 

The group G = GL(m, M.) of real non-singular m x m matrices g acts on f2 by 
the rule r — ► r g (r) = gr^' so that T gi T g2 — T gi92 ; <7i,<?2 £ G. The corresponding 
G-invariant measure on is 

(2.1) d»r = \r\- {m+l) ' 2 dr, \r\ = det(r), 

[Tl p. 18]. The group G is transitive on 17 but not simply transitive. The transi- 
tivity retains if we restrict to the subgroup T m of upper triangular matrices with 
positive diagonal elements. This subgroup is simply transitive. The subgroup of 
lower triangular matrices with positive diagonal elements also acts simply transi- 
tively on 17. Each r £ 17 has a unique representation r = t't, t = (Uj) £ T m , so 
that 

m 

(2.2) dr = 2 m Y[ t™ - j+1 dt jtj dU , dU = J[ dt L] ; 

[Tl p. 39]. An alternative representation reads r = tt' , t £ T m . To connect both 
representations, let 

" 1 



(2.3) r* = uruj, to 
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If r 6 f2 and r* = t't, t G T m , then r — tt 1 , where r = Lut'cu G T m . 

We use a standard notation 0(n) and SO(n) for the group of real orthogonal 
n x n matrices and its connected component of the identity, respectively. The 
invariant measure on SO(n) is normalized to be of total mass 1. The Schwartz 
space S = S(9Jl„ !m ) is identified with the respective space on M. nm . 

The Fourier transform of a function / G L 1 (2H„ !m ) is defined by 

(2.4) (J7)(*)= / e tr(iy ' x) /(x)dx, yeM nim . 

£Dt»,m 

This is the usual Fourier transform on R" m , and the relevant Parseval equality 
reads 

(2.5) (J7,J» = (2*)™* (/,*>), 
where 



{f,if)= J f(x)ip(x)dx. 

Lemma 2.1. (see, e.g., |Mul pp. 57-59] ). 

(i) If x = ayb, where y G 9Jl„. m , a G GL(n,R) 7 and b G GL(m,R), then dx = 
\a\ m \b\ n dy. 

(ii) If r = qsq' , where s G V m and q G GL(m,R), then dr = \q\ m+ ds. 

(iii) If r = s -1 , s G O, i/ien rGSl and dr = |s| _m_1 (is. 

2.2. The composite power function. Given r — (rij) G £1, let Ao(r) = 1, 
Ai(r) = ri,!, A2(r), . . ., A m (r) — \r\ be the corresponding principal minors which 
are strictly positive jMul p. 586]. For A = (Ai,...,A m ) G C m , the composite 
power function of the cone fl is defined by 



Ai/2 

(2-6) r x = [] 



Mr) 



(2.7) = A!(r)^ ■ ■ ■ A m _!(r) Am '^ AT " A m (r) : 



Remark 2.2. In the case m = 1, the function l|2.6|) becomes r x / 2 . This no- 
tational confusion is easily resolved if one takes into account the equality (|2.9|) 
below. 

The composite power functions associated to homogeneous cones were intro- 
duced by S. Gindikin |£3. We also refer to |FK], |Kh2] , and [T], where EH is 
written in different notation. 

We denote 

A| = Ai + • • • + A m . 

If Ai/2, . . . , A m /2 are integers and Ai > M > ■ ■ ■ > A TO , then r x is a polynomial of 
degree |A|/2. In the special case Ai = . . . = A m = A we add the subscript so that 

A = (A,...,A) (GC m ) 

and 



(2.8) 



r A = 1^1 A/2 
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If r = ft, teT m , then A,(r) = [A;(i)] 2 = ]J l j=1 1] 3 , and therefore, 

m 

(2-9) r* = l[t%=ir x (t). 

j'=i 

The expression TV\(t) is a multiplicative character of T m , so that 



(2.10) 



TT\(tlt 2 ) = 7r X (h) Ttxfa), *x(t ) = 7T_ A (t). 



For A = (Ai, . . . , A m ), let A* = (A m , . . . , Ai) be the reverse vector; (A*)j = 
Am-j+i- To each r £ we associate the matrix r* = wrw (see (|2.3|) 'l with the 
components (r*)jj = r m _,-+i, m _i+i- 



Lemma 2.3. Let A, // e C m ; re!!. TTien 

A+/x A /x A+a n = Ai ia/2 



(2.11) 

(2.12) (i'rt) A = (i'i) A r A , £ G T, 



= r , r 
'^A a 



a = (a, . . . , a); 



(2.13) 
(2.14) 



,,A, 



1\A _ -A, . 



(cr) A = cl A l/ 2 r A , (cr) A = cl A l/ 2 r A , c> 0. 



PROOF. These statements (up to notation) may be found in different sources 
Uji |FK| . |Kh2| . |T] . For the sake of completeness, we outline the proof. The 
property (|2.11|l is clear in view of l|2.9|l and (|2.8|) . To prove l|2.12|l , it suffices to set 
r = t't, t € T m , and make use of (|2~§|) . To prove (|2~T3l) . let r = ft, t e T m . Then 
by & 

m 

'' A - 1 1 ^ 

On the other hand, since r _1 = t^ 1 (f)~ 1 , and (r -1 )* = wr^'w = (wi _1 CL')(wi _1 w)', 
by taking into account the equalities 



t- 1 = 



tot u> = 



i.i 



[CTuI p. 580 (vii)], we have 



1\-A 







r t- 1 

m,m 


\ 







* 













-1 

1,1 - 




*M _ 





Y[ *m-j+l, m-j+1 II * 



A m _ 3+ i 
3,3 



3 = 1 



3 = 1 



This gives r A * = (r _1 )^ A . Replacing r by r _1 and A* by A, we obtain the second 
equality in l|2.13|l . The first equality in (|2.14|) follows immediately from l|2.6|l . The 
second equality is a consequence of the first one and l|2.13|l : 

i r -i\-\» = r \\\/2r r -i\-\ m = n \x\/2 r x 



(cr) A = (c-V" 1 ) 



□ 
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2.3. Gamma function of the cone fi. The gamma function of the cone f2 
is defined by 

(2.15) r n (A) = ( r x e-^ r U,r. 



This integral converges absolutely if and only if Re Xj > j — 1, and represents a 
product of ordinary T-functions: 

m 

(2.16) r n (A) = tt™^- 1 '/ 4 J] r((A, - j + l)/2), 

see, e.g., [FK1 p. 123]. 

Lemma 2.4. For sell and i?e Xj > j - 1, 

(2.17) J r x e- tT( - rs U*r = r n (A) s,: A * . 

h 

Proof. This equality is known ([HD P- 23 ], |FKl p. 124 ], [Kh2] L By 

it is equivalent to 

(2.18) / r^e'^^d^r = r n (A) s x . 



Assuming s — t't, t £ T m , and changing variable r = i'pi, for the left-hand side of 
(|2.18|l we obtain 

J {t'pt) x e-^i>)d*p r (A)(<'i) A = r n (A)«\ 

b 

□ 

An important particular case of l|2.15|l is the Siegel integral 

/m— 1 
\r\ x e-^d*r = t^™" 1 )/ 4 [] T(A - j/2), 

n *=° 
which converges absolutely if and only if Re A > (m — l)/2. By i|2.8[l . 

(2.20) r o (A ) = r m (A/2), A = (A,...,A). 

2.4. Stiefel manifolds. For n > m, let V n , m — {v £ 9K n ,m : t)'t> = I m } be 
the Stiefel manifold of orthonormal m-frames in R™. We fix the invariant measure 
dv on Ki.m [Mu, p. 70] normalized by 

/^m —nm j '2 
T m (n/2) 

V n , m 

and denote d*v — a~ x m dv. The polar decomposition on 97l„ !m is defined according 
to the following lemma; see, e.g., |Mul pp. 66, 591], jMaj . 

Lemma 2.5. If x <E 97l„ !m , rank(ir) = m, n>m, then 

(2.22) x = vr 1/2 , veV ntm , r = x'x £ 0, 

and dx = 2- m \r\ { - n - m -V/ 2 drdv. 



s 
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A modification of Lemma 12.51 in terms of upper triangular matrices reads as 
follows. 

Lemma 2.6. f[FJ. |Ru4p If x G 97t„, m , rank(x) = to, n > m, then 

X llt^ U G Vn,m: t G T m , 

and 

m 

dx = J^J t™~ ^ dtjjdt*dv, di* = TT^ij- 

3. The generalized zeta integrals 

By taking into account the polar decomposition x — vr 1 ^ 2 (see Lemma 12.511 . 
we introduce the following generalized zeta integrals (or zeta distributions): 



(3.1) Zi^XJ) = J r*f(v)ct>(x)dx = (r*f,<p), 

(3.2) Z*((j>, A, /) = J r*f(v)t(x)dx = (r*f,<f>). 

Wl n , m 

Here r = x'x, v = xr~ x / 2 , / is a fixed integrable function on 14,, m, and </> G §(0Jl„ !m ) 
is a test function. 

The following particular cases are worth mentioning. 

1°. A = A = (A,...,A). 

In this case 



dx. 



(3.3) Z(</>,A,/)=Z»(0,A,/) = f (detr) x / 2 f(v)<j>(x) 

For / = 1, zeta integrals of this type were studied in |Ge| . |Khlj : see also |Ru4| 
and references therein. 
2°. to = 1. 

r = x'x = \x\ 2 and (|3.3|l reads 



(3.4) Z(<j>,\,f) = I Ixff^^dx. 



Distributions of this form are well known in analysis and integral geometry; see, 
e.g., [He], Hi], jKoj, |RuT] . 

3°. A G C m , / = 1. 

This case was explored in (FK (in the context of Jordan algebras) by invoking 
the relevant ii-Bessel functions; see also [Ra| . |Kh2| . [B], BSZ , and references 
therein. 

4°. A G C m , / is a determinantlally homogeneous harmonic polynomial (see 
Section 4.2). 

In this case, integrals (|3.1|) and (|3.2|) were studied in |Clj using the argument 
close to [FT?] . 

Most of the publications mentioned above were focused on evaluation of the 
Fourier transform of the corresponding zeta distributions. This transform is realized 
in the form of the relevant functional equation of the Parseval type. 
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Our nearest goal is to investigate convergence of integrals (13. 1|) and l|3.2|) . and 
their analyticity in the A-variable. We denote 

(3.5) A = {A e C m : Re Xj > j —n-1 Vj = 1, . . . , m}, 

(3.6) A = {A e C m :Xj=j-n-l for some 

i € {1, ...,m}, and I € {1, 3, 5, . . . }}. 

Lemma 3.1. The integrals \ H. 1)) and f.V. ty) are absolutely convergent if and only 
if X E A and extend as meromorphic functions of A with the polar set Aq . The 
normalized zeta integrals 

(3.7) ^A,/) = ^A, ^A,/) = ff^, 

rn(A + n ) rn(A + n ) 

n = {n, . . . , n), are entire functions of A. 

Proof. Let us consider the integral (|3.1|l . We set x — vt, »£ V rum , t E T m , 
and make use of Lemma |2~B1 By taking into account that x'x = t't and t (t't) — x / 2 g 
0(m), owing to (|2.9p . we obtain 



3=1 



where 



F(i M , . . . ,t m , m ) = y dt* J f(v)4>(vt)dv, di*=JJdi; 



i<j 



Since i* 1 extends as an even Schwartz function in each argument, it can be written 
as 

where F € S(K m ) (use, e.g., Lemma 5.4 from |Trl p. 56]). Replacing j by Sjj, 
we represent (13.81) as a direct product of one-dimensional distributions 

m 

(3.9) Zfa A,/) = (n( s «)+ J+ "^ 1)/2 , F ( Sl)1 ,... !Sm , m )). 

i=i 

It follows that the integral (|3.1|) is absolutely convergent provided Re Xj > j — n—1, 
i.e., A e A. The condition A 6 A is strict. This claim becomes clear if we choose 
/ = 1 and 4>{x) = e _tr ^ x > which give 

(3.10) Z((f>,XJ)= J (x'x) x e- t < x ' x Ux = 2- m o n ^Tn(X + n Q ). 

Tl n , m 

Furthermore, since (sj,j)+ 3+ ™ J 2 extends as a meromorphic distribution with 
the only poles Xj = j — n — 1, j — n — 3, . . . , then, by the fundamental Hartogs 
theorem |Sh| . the function A — > Z(<j),\ 7 f) extends as a meromorphic function 
with the polar set Ao. By the same reason, a direct product of the normalized 
distributions ^AXCAj + n — j + l)/2) is an entire function of A. 
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Let us consider the integral H3.2|) . By changing variable x — yu> where u> is the 
matrix (|2.3(l . we obtain r* = [x'x)* — (ujy'yuj)* = y'y. Hence (set y = ut, u 6 



, m i 7~ G ^"m ) 5 



M<t>,\f) = J (y'y) x f(Muv'v«>)~ 1/ *)<l>(yu)dv 

97L,„ 



J 'fir,.; -.,^11-",'" ''/- 



i=i 
— + 

where, as above (note that tlo (lot'tlo)^ 1 ^ 2 G O(m)), 

$(n,i,---,7" m , TO ) = J dr * J f( u ) ( t } ( UTUJ ) du 

R m(m-l)/2 V„, m 

= $o(7f,i,...,<J, $oGS(M m ). 

This gives 

m 

z,(0,a,/) = (n( s w)+ J+ "^" 1)/2 ! *o(«u ^5), 

i=i 

and the result follows as in the previous case. □ 

4. Connection between zeta integrals and the composite cosine 

transform 

4.1. The composite cosine transform. Let A = (Ai, . . . , A m ) G C m , / be 

an integrable function on V n>m . We introduce the following family of intertwining 
operators 

(4.1) {T x f)(u)= J f{v)(u'vv'u) x dv, ueV n , m , n>m, 

v n , m 

which commute with the left action of the orthogonal group 0(n). We call (T x f)(u) 
the composite cosine transform of f. In the particular case Ai = . . . = A m = A, 
the integral lj4.1|) represents the usual A-cosine transform 

(4.2) (T X f)(u)= J f(v)\det(v'u)\ x dv. 

v n , m 

The function (T x f)(u) extends to all matrices y G 9JT n , m of rank m if we set 
y = ut, u G V n ,mi t S T m . Indeed, by l|2.12|l . 

(4.3) (T x f)(y)=r x (T x f)(u) 
where r x = (t't) x = (y'y) x is "the radial part" of (T x f)(y). 

Theorem 4.1. For f G L {y n ,m)> the integral (T x f)(u) converges absolutely 
for almost all u G V n , m if and only if Re Xj > j — m — 1 for all j = 1,2, ... , m and 
represents an analytic function of A in this domain. For such \, the linear operator 
T x is bounded on L 1 (V n , trn ). 

This statement follows immediately from the following lemma which is of in- 
dependent interest. 
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Lemma 4.2. Let u.v e V n , m . Then 



(4.4) /ee y (iiWii) A 4= J (u'vv'u) x dv 



2 m ir-/ 2 r a (A + m ) 
r m (m/2) r n (A + n ) 



TTiis integral converges absolutely if and only if Re Xj > j — m — 1 /or all j 



1,2,..., m. 



Proof. The first equality in (|4.4|) becomes clear if we write both integrals as 
those over the group SO(n) (see Lemma 2.4 in |GR| ). These integrals are, in fact, 
constant with respect to the corresponding exterior variables. Thus, one can write 

Irr 



(4.5) 



(u'vqv'qu) du, Vq 







To evaluate / we introduce an auxiliary integral 



(4.6) 



.4 



(x vqVqX) e y ' ax. 



n — m.ni- 



^ b 'v db. 



and transform it in two different ways. Let first 

a e M m ^ m , b e M 
Then v' x = a, x'x = a' a + 6'6, and we have 

(4.7) A = A X A 2 , A x = J (a'a) x e - tl( > a ' a Ua, A 2 

Tt m , m 

By Lemma 1231 (j2~TTJ) . (j2~TK|l . and lf2~2T|) . we obtain 

(4.8) A 1 = 2-™* mtm [ r^e-^M r = ^I^ + go) 

J T m (m/2) 
n 

provided Re Xj > j — m — 1, j = 1,2, ... ,m. The last condition is sharp and 
provides the "only if part in the lemma and in the Theorem 14.11 For A 2 we have 



m(n—m) 



A 2 = 



ds 



Thus 
(4.9) 



.4 



TT nm ' 2 r n (A + m ) 



_ 7r m(n-m)/2 



Re Xj > j — m — 1 . 



r m (m/2) ' 

On the other hand, by setting x = ut, u € V n<m , t G T m , owing to Lemma 12.61 



we obtain 



A= j e - tr( *' 4) dfi(t) J (t'u'v v' ut) x du, 

T m Vn.m 
m 

d^{t) = Y[ t'jj'' dtjj dt*, dt* = Y\_ dti,j- 



3 = 1 
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By (I2.12|l . one can write 

(4.10) A = BI, 

where / is our integral (|4.5(l and 

B = f {t't) x e~ tr(t,t) dn(t) = 



m . • i 1 

= 2- ra T !i (A + n ), fleA; > j-n-1. 
Combining this with l|4.9|) and (|4.1()(l . we obtain 

r o (A + m ) 



2- m T n (X + n Q )I 



r m (m/2) 

provided that i?e Aj > j — m — 1 for all j = 1, 2, . . . , m. This gives (|4.4|) . □ 
Corollary 4.3. Let Ai = . . . = A m = A, Re A > — 1. TTien 



/• 9m nm/2 p f m+A \ 

(4-ii) / idet(t/«)i^= y ( ' m n _A r - 

If we set A = 1 in l|4.11|l and replace cfc by the normalized measure d*v then 
the resulting formula bears an important geometrical meaning. Namely, let £ and 
rj be m-dimensional linear subspaces of K", i.e., £,17 S G n ,m, and let u and u be 
coordinate frames of £ and 7/, respectively. Then |det(w'u)| = [S\rj\ is the volume 
of the projection onto r\ of the parallelepiped spanned by u. The corresponding 
averaged volume is valuated as 

(4.12) / |det(t/u)|d*« = / Kl*?]dt?= ,H 

v ' J Ll/J r m (f)r m (H±i) 

4.2. The basic functional equation. Below we establish connection be- 
tween zeta integrals and composite cosine transforms. This can be done in the 
form of a functional equation which is, in fact, the usual Parseval equality in the 
framework of the corresponding Fourier analysis. We start with the following 

Lemma 4.4. Let f be an integrable O(m) right- invariant junction on V n<m ; 
x = vr 1 ' 2 , tp(x) = /(w)|r|( m - n )/ 2 . For sen and cj> € S(Wl n , m ), 

(4.13) / WWy)dy I f{v)e-^' ys ~ ly ' v Uv 

Wt n , m v ^ 

= (2n)( n - m ^ m J rp{x)e- t < X8X '/ i * ) ~$jc)dx. 

an,.™ 
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Proof. By the Parseval equality (|2.5|l . it suffices to find the Fourier transform 
of the function tp s (x) = ip(x) e~ tr ( xsx / i7l \ In polar coordinates we have 

(^ S )(y) = 2- m J f{v)dvj | r |-l/2 e tr(V l ,r 1 /=_ r V2 sr i/2 /47r) ^ 

v n , m n 
(replace v by t>7, 7 € O(m), and integrate in 7) 

= 2- m [ f(v)dv I dry ! ^-^e^W^'^'^'V*^ dr 



m O(m) n 

— I f{v)dv [ t&W**—"'/*") dz 



@m,rn J J 

The inner integral can be evaluated by the formula 

(4,4) / ^.>.-^,« & _ (2 ^ W -^ e ^- n 

where C = v'y, [Herzl p. 481]. This gives 

V„, m 

and KTty follows. □ 
Let us compute the Fourier transform of the function 

(4-15) <p x (x) = r * ' "° ' f(v) 

where x = vr 1 / 2 , no = (n, . 



. n 



Theorem 4.5. Let f be an integrable 0(m) right-invariant function on V n ^ m , 
X € C m . Then 

(4.16) { ^ x )(y) = —^ r(T A /)(2/), cx = 2-^n m ' 2 / 2 /o- m , mi 

1 q(A + mo) 

in the sense of §' -distributions, i.e. 

( 4 ' 1? ) F75TI T ( rA/ ' T ^ = ^ $ 

1 o (A + m j 

= (2^) n ™Z°(0,-A*-n o ,/) 

for each 4> £ §(9Jl nim ). 

Proof. The equality (|4.17|) demonstrates an intimate interrelation between 
zeta integrals and cosine transforms. To prove 1)4. 17|) . we multiply 1)4. 13[) by s A + m ° 
and integrate against d*s. We obtain 

1 



(4.18) / {f<t>){y)dy / f(v) A 1 {v'y) dv 

= (27r)("- m ) m J i/>{x) A 2 {x)^x) dx, 
9JL m 
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where ip{x) = f{v)\r\^ m n ^ 2 , 

Ai(C) = J S A+mo | S r m / 2 e- tr « s "«')^ S , (£Tl m , m , 
h 

A 2 (x) = f s x+mo e- tT ( X8X '/ 4n U*s, xem n , m . 

Q 

Let us compute A\ and A 2 . By (|2.11|) . setting a — ir('(, r = s , we obtain 

A x (a) = y*(r- 1 ) A e- tr(ar) d >t r 
a 

/" r ~ A *e- tr(ar) d„r (r. = wrw; sec STto ) 

(4.19) = |r- A 'e- tr ( a ' r »ir'Pr !i (-A,)« A 

h 

provided i?e (A*)j = Re A m -j+i < 1 — j, or i?e Aj < j — m. Finally, by 12.14|l . we 
have 

A 1 (C)=7rl*l/ 2 r n (-A„)(C'0 A i ReXjKj-m. 
Furthermore, by (|2.17fl and l|2.14[) . 

A 2 (x) = (4nf^ +m2 ^ 2 T n (\ + m )(x'x)- x '- mo , ReXj > j - m - 1. 
Hence, (|4.18l) reads 

( 4 - 2 °) r-7TT — V / (^)(y)(T x f)(y)dy 

Om, m I n(A + mo) J 



(27r) (n ~ m)m (47r) (|A|+m2)/2 

r n (-A„) 

9JI 



(4.21) = (2^)("- m ) m (4^)d A l+ m2 )/ 2 Z°(</>, -A, - n , /), 



(4.22) j — m — 1 < i?e Aj < j - m. 

By Lemma [3.11 the expression (|4.21|) extends as an entire function of A. Hence, 
it can be regarded as analytic continuation of the integral l|4.20|l outside of the 
domain l|4.22|l . and the result follows. □ 

Example 4.6. Let m > 1, Ai = . . . = A m = A, \x\ m = &et(x' x) 1 / 2 . Then 

= r!!(-A/2) fW x ' x )~ 1/2 )> ( TX f)(y)= J f(v)\det(v'y)\ x dv, 

v n , m 
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and we have 

o — Am _m 2 /2 

(4.23) (F<Px)(y) = r 77T- W9 , (T x f)(y). 

ffm,™r ra ((A + m)/2) 

Example 4.7. Let m = 1. Then for i.i/el"\ {0}, 
In this case 

(4.24) ^)^ = r 2 ( r+I)T2) (rA/)(y) - 

4.3. The case of homogeneous polynomials. Let Pk(x) be a polynomial 
on 9Jl,i. m which is harmonic (as a function on M nm ) and determinantally homoge- 
neous of degree k, so that 

(4.25) P k (xg) = det(g) k P k {x), Mg G GL(m,R), x G 9Jt„ >m . 

The latter means that Pfc is a usual homogeneous harmonic polynomial of degree 
km on R" m . Theorem 14.51 can be essentially strengthened if we choose / to be the 
restriction of Pk(x) onto V n , m - According to l|4.15|l . for x — vr 1 ! 2 , we denote 

~— A*— no 

(4-26) ip x , k {x) = * P k (v). 

1 q(— A* J 

Theorem 4.8. Le£ y = vr 1 / 2 , v G K,m, re!l. For a// A G C m , 
in the sense of §' -distributions, i.e., 

rf x r n (ko-X) A) = 

rn(A + k + n ) 

= (2^r™Z«(^-A.-n ,P fe ) 

/or eac/i (/> G S(9Jt„ )m ). 

Proof. By the Hecke identity |StJ p. 87], 

(4.28) J P k {x)e- tI ^ x ' x) e tl( - 2 ' riv '^ dx ^ i km P^e'^y'^, 

Tt n , m 

or (replace x by xs 1 ^ 2 /2tt and 1/ by ys" 1 / 2 , s G f2) 
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By the Parseval equality, 
(4.29) \s\- k - n ' 2 I PkMe-^'-WjWfidy 



m n , m 

(2ni)~ km [ P k (x)e~ tl{xsx ' /4 ^&xjdx. 



9K„, m 

We multiply l|4.29|) by s A + n o+ k o an d integrate against d*s. This gives 



P k (y)h(y) mv) dy = {2m)~ km J P k (x)I 2 (x) cf>{x) dx, 
where 



n n 
As in the proof of Theorem 14.51 we have 

h(y) = TT^- km ^ 2 T n (k - A,)(y'y) A - ko , Re Xj < j + k - i 

and 

h{x) = (4^)(l A l+" m + fcm )/ 2 r (A + n + k ){x'x)- x *-^-^\ 
Re Xj > j — n — k — 1 . 
Hence, if j — n — k — 1 < i?e A 3 < j + k — m, then 



(4.30) r n (k -A*) / P fc (y)(^) A - k0 ^(2/)dy 

= c A r n (A + ko+no) /" P fe (aO(z'aO; A *- n °- k °^)dz, 
2Jt„, m 

c\ = i-krn2nm+\x\ 7r nm/2 _ By taking into account homogeneity of P k (i.e., P^vr 1 / 2 ) 
P k (v) |r| fc / 2 ), and setting tf)\ t k( x ) = Pk( v ) rX , x — vr 1 / 2 , this can be written as 



rn(-A*)r n (A + ko+n ) 
(4.31) = (2n) nm Z°(0,-K-n a ,P k ), 



(4.32) j - n - k - 1 < Re Xj < j + k - m, 

d x = 2-\ x W mn l 2 i krn . Since the domain is not void for all fc = 0, 1, 2, . . and 
the normalized zeta integral l|4.31|) is an entire function of A (use Lemma 13.10 . the 
result follows. □ 

Example 4.9. Let m > 1, Ai = . . . = X m = X, 

PxAx) = r'I^A/2) PkMx'x)- 1 ' 2 ). 
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Then 



(4.33) (^x, k )(y) = = , tii {k ((x X l{ 2 l w 9 , \y\ x m P k (y(y'y)- 1/2 ), 

r m (-A/2) r m ((A+fc+n)/2) 



d x = 2- Xm TT nm/2 i km . 
Example 4.10. Let m = 1, 



I X I ^ ^ / X \ 



r(-A/2) K V|*|. 

Then 

d A = 2- A 7r n / 2 i fc . 

5. Injectivity of the composite cosine transform 

We denote by £ the set of all A = (Ai, . . . , A m ) S C™ satisfying i?e Aj > j— to— 1 
for all j = 1, ...,m. By Theorem 14.11 (T x f)(v) exists a.e. on V^ im for every 
/ € i 1 (V r „. m ) if and only if A 6 £. In the following we focus only on this domain, 
although other A can also be treated using analytic continuation. We also restrict 
our consideration to the space L iy n ,m) which consists of 0(m) right-invariant 
integrable functions on V n:Tn . This space is isomorphic to the space L 1 (G niTn ) of 
integrable functions on the Grassmann manifold G n ,m- 

Theorem 5.1. Let n > to and A = (Ai, . . . A m ) G £. // 

(5.1) Aj+m-j^ 0,2,4, ... for all j = 1, ...,m, 

then the operator 

(T x f)(u) = [ f(v){u'vv'u) x dv 



is injective on L^(V n ^ m ). If 2m < n and \5.1]) fails then T x is non-injective. 

To prove this theorem, minor preparation is still needed. 

Definition 5.2. Following Herz , we call a polynomial Pk{x) on 9Jt„. m an H- 
polynomial of degree k if it is O(m) right-invariant, harmonic, and determinantally 
homogeneous of degree k. We denote by Ti k the space of all such polynomials. 

Lemma 5.3. Let P k e Ti k , 

(5.2) W (A) = rTx^fxfr+nV Aecm 

rn(- A» ) T n {\ + k + n ) 
If X does not belong to the polar set ofT^(X + mo), then 

(5.3) {T x P k ){vr 1 l 2 ) = c^ k {X)P k {v)r x , c = ir m ^ m ^ 2 i km a m ^ 
in the sense of §' -distributions. 
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PROOF. Let us compare (14.16)1 and l|4.27(l . assuming / to be the restriction of 
Pfc onto V n , m - For all A G C m , we obtain 

r^(A + m ) rn(-A»)rn(A + ko + n ) 

in the sense of S'-distributions. If we exclude all A belonging to the polar set of 
r n (A + mo), then wc get |Q. □ 

Corollary 5.4. If\e£ and P k G Hu, then for all v G V n , m , 

(5.5) {T x P k ){v) = c^ k {\)P k {v), 
c and /ife(A) having the same meaning as in 

Proof. The function 

r fi (A + mo) = J] r( Aj + m 2 ' J + 1 ) 

has no poles in the domain £, and therefore, by 1)5. 3)1 . 

(5.6) ((T A Pfc)(wr 1 / 2 ), 0) = c^tfc(A) (P k (v)r x , <f) 

for all (/>(y) = ipivr 1 ^ 2 ) G S(9Jt„. m ). Now we choose <f>(y) = x{ r )i ) { v ) where x( r ) 1S 
a non-negative C°° cut-off function supported away from the boundary of f2 and 
ijj(v) is a C°° function on V n: , n . By passing to polar coordinates, from ()5.6I) and 
(14.31) we obtain 



[{T x P k )(v) - cMfc(A) P fe («)] ^(v) cfo = 0, c x = const ^ 0. 

This implies H5.5jl . □ 

Remark 5.5. A simple computation shows that for k = 0, the equality 1)5.4)1 
coincides with 1)4. 4[l . Note also that if m = 1 then P/. g 7Y fc is necessarily of even 
degree, and we have 

r(±+±)r(^) 

(5.7) (T x P k )(v) = 2^'\-l)^ y ) \ 2 J P k (v). 

r (- I) r( A i|±^J 

This coincides with the known formula (|1.4() for spherical harmonics. 

Remark 5.6. An important question is, do there exist ^-polynomials of a 
given degree k? Note that for n = m we have exactly two such polynomials, 
namely, Po{x) = 1 and P\(x) — det(x). The following statement is due to Herz 
Herz], p. 484: For 2m < n there exist H -polynomials of every degree k. It is also 
known [TT] . p. 27, that for 2m < n, the space TL k is spanned by polynomials of 
the form P k (x) = det(a'x) fc where a is a complex n x m matrix satisfying a' a = 0. 

Proof of Theorem 15.11 To prove the first statement, we consider the equality 

(5.8) ((T x f)(vr^ 2 ), T$ = A(X) (r^^ f(v), 0), 

(27rr m ro(A + m ) 

^ > ~ fw — — ' 
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which follows from Ij4.17|l and Ij4.15|l . Suppose that (T x f)(v) = almost everywhere 
on V n>m for some A e £. Then (T x f)(y), y = vr 1 / 2 G Tl n<m , is zero for almost 
all y G m n , m , and ^ yields A(X) (r-- A *- n ° f(v), (j)) = 0. The assumption ij^Tjl 
together with A G £ imply 

Re Xj > j — m — 1 and Xj ^ j — m, j — m + 2, . . . . 

Hence A is not a pole of ro(— A*), and therefore A(X) ^ 0. This gives 

(5.9) (r- X '- no f(v), <f>) = 0, 

the left-hand side being understood in the sense of analytic continuation. By choos- 
ing (j> as in the proof of Corollary 15.41 we obtain f(v) = a.e. on V n ,m- 

Let us prove the second statement of the theorem. We first note that for 
2m < n, i7-polynomials of every degree k do exist. Hence we can proceed as 
follows. We observe that the function rn(ko — A*) in the expression of /Ufe(A) has 
the form 

ro(m-l)/4 1~T r ( k - - j + 1 

m(m-l)/4 TT r / ^ + 3 ' ~ ^ j ~ TO 



ro(ko-A,) = 1)/4 n r v 



7T 



(we remind that (A*)j = J?eA m — i+i)' It has no poles in £ if 
(5.10) k > Xj + m — j for all j = 1, 2, . . . , m. 

Since ro(A+mo) also has no poles in £, then, by (|5.5|l . T x Pk = for all A; satisfying 



(|5.10|l . provided that A is a pole of r$i(— A*) (i.e., the condition l|5.1|l fails). Thus 
T x is non-injective in this case. □ 
We do not know if the condition l|5.1|l is necessary for injectivity of T x in 
the case 2m > n. We suspect that to answer this question one has to explore 
T x on polynomial representations of the group SO(n) parameterized by highest 
weights (mi, m2, ■ ■ ■ , wrn/2]); more general than just (fc, k, . . . , k) corresponding to 
Hfe-spaces; cf. jStrj . |TT| . However, if Ai = •• • = A m = A then for the A-cosine 
transform 

(T x f)(u)= J f(v)\det(v'u)\ x dv (or (T A /)(0 = J f (v) Wtf d V ) , 

V n , m G„, m 

we can give a complete answer. 

Theorem 5.7. Let n > m and ReX > — 1. Then T x is injective on L (y n ,m) 
(or on L 1 (G„ jm )) if and only if A ^ 0, 1, 2, 

PROOF. If Ai = ... X m = X then the condition A ^ 0, 1, 2, . . . coincides with 
(|5.1|) . Hence, for 2m < n, the result follows from Theorem 15.11 If 2m > 77, i.e., 
2(?7 — m) < 77, we reduce the problem to the dual one on the manifold V„, n _ m . To 
this end we replace the integral T x f by the equivalent expression in terms of G„ !m 
and write the latter as an integral over the dual Grassmann manifold G„.„- m . This 
argument can be realized as follows. 

Let £ G G ntm and 77 € G n ^ m be m-dimensional subspaces of K™ spanned u G 
V^ !m and v G V"„. m , respectively. We denote by £ G G„.„_ m and Ty -1 G G n , n -m the 
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corresponding orthogonal complements, and choose coordinate frames u± G V n , n -m 
in f and v± G V n ,n-m in f] • By using the same notation as in l|1.6fl . we have 

(5.H) Hi\ = 

Indeed (we recall that | • | denotes the determinant of the matrix inside) 

M£] 2 = det(Vw)| 2 = \u'vv'u\ = \u'(I n — v±v'j_)u\ — \I n — u'v±v' ± u\ 

= \In-m - v' ± Uu'v±\ = \I„- m ~ v' x (I n - U±u' x )v±\ 

= \v'j_u±u'j_v±\ = |dct(u^_u^)| 2 

Now we successively define the functions F(rj) on G n<m , F±(i] ± ) on G n ,„_ m , and 
f±(v±) on V n ,n — 7n 

by 

F( V ) = f(u), F ± (^) = F(( V ^), f ± (v ± ) = F ± tf). 

Then 

(T x f)(u) = J F(r,){ V \Z] x dr, 

G„,m 

F±(v X )[rr L \t X ] X dTi 1 - 



f±(v ± )\det(v' ± u ± )\ x dv± 



= (T£f±)(u ± ). 

Thus T x can be expressed as the similar operator on the dual manifold V n ,n-m- 
By above, the latter is non-injective for 2(n — m) < n, and we are done. □ 
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